The Institutional Dimension of Document Quality Judgments
نویسندگان
چکیده
In addition to relevance, there are other factors that contribute to the utility of a document. For examples, content properties like depth of analysis and multiplicity of viewpoints, and presentational properties like readability and verbosity, all will affect the usefulness of a document. These kinds of relevance-independent properties are difficult to determine, as their estimations are more likely to be affected by personal aspects of one's knowledge structure. Reliability of judgments on those properties may decrease when one moves from the personal level to the global level. In this paper, we report several experiments on document qualities. In our experiments, we explore the correlation of judgments on nine different document qualities to see if we can generate fewer dimensions underlying the judgments. We also investigate the issue of reliability of the judgments and discuss its theoretical implications. We find that, between the global level of agreement of judgment and the inter-personal level of agreement of judgment, there is an intermediate level of agreement, which can be characterized as an inter-institutional level.
منابع مشابه
Institutional Requirements for Enhancing Financial Stability in Iran
The conditions in which the financial system can continuously provide the necessary services for the proper functioning of the economy is called “financial stability”. In this study, using global experience and Iranian data from 1990 to 2016, an aggregate index of financial stability for Iran has been constructed under the four dimensions of "deposit taker institutions", "macroeconomics", "for...
متن کاملCrowdsourcing Relevance Assessments: The Unexpected Benefits of Limiting the Time to Judge
Crowdsourcing has become an alternative approach to collect relevance judgments at scale thanks to the availability of crowdsourcing platforms and quality control techniques that allow to obtain reliable results. Previous work has used crowdsourcing to ask multiple crowd workers to judge the relevance of a document with respect to a query and studied how to best aggregate multiple judgments of ...
متن کاملThe Role of Institutional Quality in Forecasting the Total Stock Price Index: Case Study of Developing and Developed Countries
The purpose of this paper is the empirically examination of the effect of the Institutional Quality Components (Voice and Accountability, political stability and Absence of Violence and Terrorism, government effectiveness, regulatory quality, rule of law and control of corruption) on the prediction of total stock price index. The study was carried out using data extracted from 53 countries from...
متن کاملInflation Bias, Time Inconsistency of Monetary and Fiscal Policies and Institutional Quality
In developing countries, weak institutional quality can increase the probability of applying discretionary policies and can have a great impact on their double-digit inflation. Surico (2008) calculated inflation bias, but he considered just monetary policy and he did not pay attention to the institutions. Therefore, we design a model which considers the discretion in monetary and fiscal policie...
متن کاملManaging the Quality of Large-Scale Crowdsourcing
Crowdsourcing can be used to obtain relevance judgments needed for the evaluation of information retrieval systems. However, the quality of crowdsourced relevance judgments may be questionable; a substantial amount of workers appear to spam HITs in order to maximize their hourly wages, and workers may know less than expert annotators about the topic being queried. The task for the TREC 2011 Cro...
متن کامل